1. Field of the Invention
The invention generally relates to the field of streaming data over a network and more specifically, the automating of business processes driven by real-time information.
2. Description of Related Art
Many integration systems for streaming and static data over a computer network or over the Internet exist already. Approaches may be found within several areas of integration middleware: extract/transform/load (ETL), enterprise application integration (EAI), enterprise information integration (EII), and domain-specific middleware like a financial market-data system or RFID middleware.
In the last decade, client server systems for static data integration have increasingly favored locating logic, to the extent possible, on the server rather than the client. Loading logic on the server lowers costs, shortens time to market, and reduces long-term risks. Integration vendors have continued this trend by introducing products that perform more integration on application servers and leave less to developers of client applications.
Current data integration systems do not provide for streaming data the same quality available for static data, nor for system integration of mixtures of streaming and static. Currently available integration products for streaming data force application developers to implement a great deal of logic related to content acquisition and transformation. This logic implementation costs application developers a great deal of time and money in both the short and long term. Moreover, to the extent that some streaming data integration products try to simplify the developer's task, such simplification is achieved by putting logic into the API (Application Programming Interface). However, putting logic in the API forces developers to use proprietary APIs and proprietary APIs become harder to maintain on a variety of platforms as the APIs evolve to incorporate more business logic.
What is needed is a flexible way for an integration system to acquire, transform, and distribute streaming data, as well as a means for the integration system to support standards (e.g. Java Message Service (JMS)). What is also needed is a means to shift the burden of content-integration from the client-side application to the server infrastructure.
Financial services firms have information intensive business processes that consume information from a large variety of sources, both internal and external. Much of the data they consume is related to activity in the financial markets and therefore updates continuously while the markets are open. These firms also generate information and distribute it within their enterprise and with external business partners.
A major financial services firm will develop hundreds or thousands of different software applications that consume, manipulate and perhaps publish data from one or more data sources. Each of these applications is designed to perform some specific business function ranging from graphical display for users to fully automated algorithmic trading.
Financial services application are typically complex, by virtue of interaction with multiple streaming and static data sources, as well as the rapid update of continually changing information. A large-scale application, for example, may consume data relating to hundreds of thousands of different financial instruments and absorb tens or hundreds of thousands of updates per second. Currently, proprietary technology (data integration systems) meets the specialized need of the industry, requiring high development costs; poor integration of the proprietary with the larger mainstream is a chronic problem.
Financial services firms must constantly improve their ability to exploit market place opportunity to remain competitive. Streaming data integration is critical. The ability of developers to rapidly provide new software to serve a data integration system is a vital aspect of preserving productivity and shortening time to market. What is needed is a standard data integration system permitting rapid product development and seamless integration with other data integration systems.
What is needed is a system and method for integration of streaming data that provides data virtualization, flexible transformation, customer-controlled (as opposed to vendor dictated or “proprietary”) data models, and standards-based interfaces. Further needed are a method and system of simultaneously integrating multiple, disparate data sources, both streaming and static, as well as executing high-through-put, low latency deployment of data to subscribers (clients applications). And a data integration system providing dynamic tailorability of subscription permissions and improved resource allocation in service of deployment are needed.